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The project described herein represents an atteapt by 
th0 0. S. Office of Education to take a acre active role in the 
national disseaination and replication of those prograas which have 
achieved the greatest success in producing reliably aeasured 
achieveaent benefits in reading and aatheaatical skills. Rith this 
general objective in aind, the a. S. Office of Education's Office of 
Planning, Budgeting, and Evaluation initiated a aultiyear 
investigation of the feasibility of packaging exeaplary projects in 
sufficient depth and detail so that the packages theaselves vould 
constitute a viable neans of replicating successful practices. The 
first year of the total investigation vas concerned with the 
developaent of packaging concepts, and the subseguent selection and 
packaging of six projects for field tryout purposes. This work has 
been accoaplished and is the subject of the present report. Four 
aajor tasks were involved in the developaent of Project Inforaation 
Packages. They weres <1) to develop criteria for choosing operational 
approaches to coapensatory education worthy of acre widespread use 
and aaenable to installation in other schools; (2) to design a 
project aodel and to decide, in general, what kinds of inforaation a 
package should contain and what the aedia should be; (3) to choose up 
to eight coapensatory education projects for packaging and develop 
project Bodeis for thea; and (4) to prepare packages for the selected 
projects. (Author/JH) 
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EXECUTIVE SUMMARY 



Since the first substantial efforts in conQ>ensatory education began 
some 10 to 15 years ago, the federal govemo^nt has spent billions of 
dollars for special programs to serve disadvantaged children. Unfortun-^ 
atelyt many of these progran^ have failed to benefit their participants^ 
and those which have been successful have not been widely adopted* The 
project described herein represents an attes^t by the U« S, Office of 
Education to take a mora active role in the national dissemination and 
replication of those programs which have achieved the greatest success 
in producing reliably measured achievement benefits in reading and math- 
ematical skills. 

With this general objective in mind» the U. S. Office of Education^ 
Office of Planning, Budgetingt and Evaluation (OPBE) initiated a multi- 
year investigation of the feasibility of packaging exemplary projects in 
sufficient depth and detail so that the packages themselves would con- 
stitute a viable means of replicating successful practices. The first 
year of the total investigation was concerned with the development of 
packaging concepts, and the subsequent selection and packaging of six 
projects for field tryout purposes. This work has been accomplished and 
is the subject of the present report. The packages are scheduled for 
tryout and evaluation at from two to five sites each during the 1974-75 
and 1975-76 school years. 

Four major tasks were involved in the development of Project Infor- 
mation Packages (PIPs). They were: fa) to develop criteria for choosing 
operational approaches to compensatory education worthy of im>re widespread 
use and amenable to installation in other schools, (b) to design a project 
model and to decide, in general, what kinds of information a Project 
Information Package should contain and what the media should be, (c) to 
choose up to eight compensatory education projects for packaging and 
develop project models for them, and (d) to prepare Project Information 
Packages for the selected projects. 
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Criteria Developiaeafc 

Project selection criteria were developed jointly by the RMC project 
team and concerned individuals within GPr«E in the four primary areas of 
effectiveness » cost, availability, and replicability* The latter two of 
these criterion areas were somewhat ^*soft*^ and judgmental. Projects were 
rejected on the ground of availability if they were unwilling to cooper- 
ate or if their selection would amount to a U.S.O.E* endorsement of 
a single publisher's or manufacturer's commercial productCs). They 
were rejected for replicability reasons only if they required resources 
nut generally available in typical school districts. £xanq>les of projects 
rejected on replicability grounds include a university-operated elementary 
school i;nd a project requiring major architectural !iH>difications to the 
typical school building* 

A recurring cost asK>unting to $475 per pupil was eventually estab- 
lished as the upper limit for project selection with the additional pro- 
vision that start-up costs not exceed $1,000 per pupil. One interesting 
fact which came to light with respect to cost was that per-pupil expendi- 
tures varied substantially as a function of the way a project was config- 
ured in a particular school district. While all six projects met the 
cost criterifib they could not always be set up as efficiently in the rep- 
lication sites and per-pupll costs, in some instances, substantially 
exceeded those of the originating sites. 

The effectiveness criterion had t^ separate aspects, statistical 
and educational significance. Both vmte cast in terms of (a) the pre***- to 
posttest gains of project participants and (b) the gains which would have 
been expected had they not received the special treatment. The educational 
significance criterion which was agreed upon specified that observed gains 
had to exceed expected gains by at least one-third standard deviation 
with respect to the national norms. The statistical significance criterion 
specified that the difference between observed and expected gains had 
to reach or exceed the five percent confidence level using a one-tailed 
statistical test. 
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Establishing the project selection criteria was a coioparatively 
simple task* Determining whether or not projects met them proved extremely 
difficulty particularly in the effectiveness area. Not one of the several 
hundred project evaluations which were exaained provided acceptable evi-* 
dence regarding project isq;>act. Even where adequate data had been collec- 
ted, analysis and reporting practices were such that no valid inferences 
could be drawn » and it was necessary to reanalyze raw data. This situa- 
tion pointed rather clearly to the need for developing a systematic ap- 
proach for validating project effects and led ultimately to the publica*^ 
tion of a report entitled A Procedural Qaide for Validating Achievement 
Gains in Educational Projects (Tallmadge & Horst , 1974) . The heart - 
the report is a 23-step "'decision tree** which leads the reader through 
a systematic consideration of factors relevant to the particular evali<- 
at ion model for which validation is sought* 

PIP Specifications and Model Development 

Preliminary planning regarding packaging concepts for PIPs was 
initiated immediately upon contract award. It was not until the first 
exemplary project had been identified, however, that the feasibility and 
attractiveness of various alternative approaches could be assessed in 
a reaI<*^world context. The final FIP configuration was arrived at through 
a sort of trial-and-^error process with each successive iteration repre- 
senting what was felt to be a significant improvement over its predecessor. 
It was several im>nths before a model design consisting of nine components 
was decided upon. The first four components were concerned with the 
planning and other pre-implementation tasks required to equip, staff, and 
set up a new project in a school district. These components included 
(a) Starter Set: Planning, (b) Project Management Directory, (c) Project 
Management Displays^ and (d) Staff Qualifications and Preparation Set. 
They were primarily intended to assist the director in installing the 
adopted project. 

The remaining five components were more specifically concerned with 
project implementation and were directly related to the day-to-day operation 
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of the project after it was installed. They Included (a) Starter Set: 
Implen^ntation» (b) Classroom Hanagec^nt Directory » (c) Student Relation-* 
ships AlbuTOi (a) Professional Relationships Guide, and (e) Hardware/Soft- 
ware Packet. 

Detailed specifications were worked out for the contents of each 
PIP cos^onent including format and mediation considerations. Some modi- 
fications were subsequently required to accommodate the unique character- 
istics of the different projects which were packaged* In most instances, 
however 9 the modifications were minor and, in no case, was it either nec- 
essary or appropriate to deviate from the basic nine-component configura- 
tion* 

The specifications for the FIPs were related to packaging methods 
and madia and reflected content considerations in only the most general 
terms. It was also necessary, therefore, to develop specifications for 
the content of each PIP based on detailed analysis of the individual 
projects being packaged » These specifications, called project models, 
described in great depth the specific topics, illustrative examples, cur- 
riculum materials, management strategies, schedules, and other items to 
be included in each component and subcomponent of each PIP* They, to-- 
gather with the packaging specifications, served as a work statement 
for Learning Achievement Corporation of San Jose, California, the sub- 
contractor to RMC responsible for the actual physical packaging of the 
selected exemplary projects. 

Exemplar y Pro j ect Selection 

Some 2,000 projects were considered at least superficially as po- 
tential candidates for packaging. Initial screening reduced this number 
to 136, and adequate information could be obtained for only 103 of these 
candidates. In-depth analysis led to rejection of 97 of the remaining 
candidates, with inadequate evidence of effectiveness accounting for 34% 
of them. 

Although every effort was made to find more, only six projects could 
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be found which t&et all of the escabllshed selection criteria. These pro- 
jects were : (a) Catch"Up t Newport Beach, California^ (b) Conquest . East 
St. Louis, Illinois, (c) Programed Tutorial Reading . Davis County, Utah, 
W) Intensive Reading Instructional Teams , Hartford, Connecticut, (e) 
High Intensity Tutoring . Highland Park, Michigan, and (f) R"3 . San Jose, 
California. 

PIP Development 

Development of the first or prototype PIP proceeded concurrently 
with the packaging specification and model design tasks. This effort 
provided, in effect, a test bed for es^rging ideas. A considerable 
amount of empirical tryout and revision occurred during the early months 
of the contract period. Four-and-one-half ufflnths after contract award, 
an Advisory Panel consisting of principals, teachers, and "experts" in 
conq>ensatory education met along with representatives of U.S.O.E. and the 
National Advisory Council for the Education of Disadvantaged Children to 
review draft materials. Two subsequent review meetings were held two and 
five months later, respectively. These three meetings, which provided 
knowledgeable inputs representing a variety of consumer viewpoints, were 
extremely helpful and contributed greatly to the final PIP design. 

The prototype PIP, along with the packaging specifications and 
project, nuadels helped to guide the development of subsequent PIPs. Al- 
though some delays were encotmtered which slgnf leant ly reduced the time 
available for planning, grants were made to 19 school districts agreeing 
to try out the PIPs during the 1974-75 school year. PIPs were delivered 
to the districts and the tryout is currently underway. Both a process 
and an outcome evaluation of this tryout are being directed by the 
Stanford Research Institute under contract with U.S.O.E. 's Office of 
Planning, Budgeting, and Evaluation. 
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1. OVERVIEW OF THE PROJECT 



The overall objective of the research and developtsent contract des- 
cribed herein was to identify effective approaches for educating disad- 
vantaged children and package them in such a viy as to facilitate their 
adoption by schools where current practices were less successful. The 
first task, as described in the Work Statement prepared by the U.S. Office 
of Education's Office of Planning, Budgeting and Evaluation (OPBE), was 
"to develop criteria for choosing operational approaches to compensatory 
education worthy of more widespread use and amenable to installation in 
other schools." The criteria, furthermore, were to help accomplish this 
objective "with a minimum of arbitrariness." 

The Request for Proposals raised a number of iasues relevant to 
project selection criteria. These issues Included the validity of var- 
ious types of gain scores, the magnitude of gain required for educational 
significance, the chance probability level acceptable for statistical 
significance, the costs associated with project replication, etc.. What 
was not immediately clear, but became obvious upon reflection, was that 
the list of issues was not limited to criteria but also included pro" 
cedures for determining whether or not the criteria were met. 

The distinction between establishing criteria and determining whether 
or not they are met is an inqjortant one. The criteria themselves are 
arbitrary and can only be chosen, while the verification procedures must 
be scientifically rigorous . Thus it is appropriate to choose a chance 
probability level which will be accepted as adequate evidence of statis- 
tical significance but, once the selection is made, rigid procedures 
must be followed to determine whether the selected criterion is met. 

Development of the project selection criteria began immediately 
upon contract award to RMC Research Corporation of Bethesda, Maryland and 
Los Altos, California. The initially established criteria required sub- 
stantial modification during the early months of the contract period. 



however, as the selection process itself revealed that many of them were 
unrealistic. Other changes were required as overlaps with other O.E. and 
H*E.W. studier> came to light and whole groups of potential candidace pro** 
Jects had to be excluded. The final list of criteria which is presented 
below was far more restrictive Chan had been suggested by the original 
Work Statement and constituted one of several reasons why the search for 
exemplary projects became a major undertaking. The final set of criteria 
was as follows: 

Relevance . Projects serving underachieving poor children in grades 
K through 12 aimed at producing cognitive achleven^nt benefits in 
reading and/or math. 

Availability . Ability of investigators to obtain enough information 
to validate the project's success and analyze it in sufficient 
depth. 

Accessibility . Documentation of procedures » results, and 
costs available; personnel cooperative; can be visited 
for validation 

Acceptability . Conformity to Office of Education policy on 
dissemination; operational in public schools; not pri- 
marily a single, commercial product. 

Cost . Recurring costs under $400 per pupil (subsequently modified 
to $475) plus start-up costs not to exceed $1,000 per pupil. 

Replicability . Major components of personnel, materials, hardware, 
and environments can be duplicated. Development of major hardware, 
facilities, or training institutions not needed. 

Effectiveness . At least two ^^Instances**^ showing evidence of edu- 
cationally and statistically significant effects on achievement. 

Educational slffllflcance . Achievement gains at least one- 
third of a standard deviation greater than expectations 
based on national norms or control group scores. 



1. An Instance, here, may be defined as a single-group evaluation. More 
than one such instance must have shown positive results before a project 
would be considered. ^ ^ 
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statistical significance > No more than one chance in 20 
(p < .05} that the observed gains could be due to chance. 

Support by UtS.O.E. funds . The project must be, or previously have 
been, the recipient of funds from U.S.O.E* through E.S.E.A. Title 
I, Title Right-^tO'^Read or other program. 

Conformance with federal regulations . Operation of the project must 
be in conformance with the pertinent program regulations and per- 
formance criteria. 

Assuming that a sufficient number of projects could be found which 
met all of the above criteria, consideration would also be given to: 

Variety . Difference in instructional strategy, breadth of target 
population served, subject matter focus, etc.. 

Finally, Bilingual progranis. Follow Through programs, and programs 
sponsored by the Bureau of Indian Affairs were specifically excluded from 
the study. 

In most cases it was a relatively straightforward matter to deter«* 
mine whether or not these criteria were met« although convenient infor- 
mation sources were often not available. Cost and effectiveness were 
exceptions and proved difficult to deal with. 

The cost problems were threefold: first, cost accounting practices 
at the project level are often inadequate and highly individualistic; 
second, alternative configurations of some projects could result in widely 
differing per-pupil expenditures; and third, regional differences in 
salary scales and other resource costs constitute a major source of varia-- 
tlon. Because the cost criterion was not considered of major importance 
(as long as costs were "reasonable" and within the limits set by the cri- 
terion) , there was not a great deal of time and effort spent refining 
cost estimates. What were felt to be reasonable approximations were used 
for selection purposes, although more detailed cost analyses were under- 
taken in order to estimate replication costs for those projects which were 
ultimately selected. 

Effectiveness considerations were given much more careful attention, 
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as endorsement of an unsuccessful project would have defeated the entire 
purpose of the research and development effort. It becainf» clear at the 
very outset of the study, however, that this kind of error would be very 
easy to make unless great care were exercised in the selection process. 

The "validation" of project effectiveness had necessarily r.o depend 
on existing evaluation data as the work schedule and budget did not per- 
mit any firsthand testing. Preliminary review of available evaluation 
reports quickly revealed that designs were tremendously varied, that 
many serious errors had been made in implementation, and that plausible 
alternative hypotheses regarding obtained results were inadequately con- 
sidered. It was clear that a systematic validation procedure was needed 
to facilitate the review and selection process. 

The classic work of Campbell and Stanley (1963) provided much of 
the informational input used in developing the validation' procedure. 
These authors, however, did not consider norm-referenced evaluation 
models — models which today are far mote widely used than any others. 

The most commonly encountered norm-referenced evaluations make use 
of grade-equivalent scores and typically express results in terms of months 
of grade-equivalent gain per month of project participation. Other norm- 
referenced evaluations make use of percentiles, deciles, or quartiles 
and assess project impact in terms of shifts from pre- to posttest. 
No serious and thorough analysis of these models and the pitfalls assoc- 
iated with their use could be found, and it was thus necessary to inves- 
tigate these issues before the validation procedure could be completed. 
Again, chia methodological task, while essential to successful execution 
of the contract effort, was largely unanticipated in the Work Statement. 

Many serious pitfalls were found to be associated with the use of 
norm-referenced evaluation models. Furthermore, the use of grade-equiva- 
lent scores in conjunction ^ith any^ type of evaluation was found virtually 
to preclude the drawing of valid inferences regarding project success. 
These findings, combined with the twelve "factors jeopardizing the internal 
and external validity of experiments" discussed by Campbell and Stanley 
(1963), were arranged in decision-tree format to provide the needed pro- 
cedural guide for validating project effectiveness (Tallmadge & Horst, 1974). 
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Concurrently with the development of project selection criteria and 
validation procedures, the search £or successful projects was initiated. 
Several lists of projects which one study or another had labeled as ex- 
emplary were provided by the U.S.O.E. Project Officer. The expectation, 
however, was that one list of 23 projects which had been assembled from 
other sources based on what was judged to be particularly convincing 
evidence would provide up to eight projects which would stand up under 
the scrutiny of the validation procedure. This expectation, too, proved 
unfounded. 

Examination of all of the projects on all of the lists did not pro- 
duce eight which met the established criteria. Additional nominations 
were obtained from State Departments of Education, from U.S.O.E. program 
offices, from consultants who participated in other phases of the study, 
and through personal professional contacts of members of the RMC staff. 
After seven months, further searching was abandoned. Only the six pro- 
jects listed below could be found which satisfied the selection criteria. 

1. Project Catch-Up , Newport Beach, California. 

Reading and math, grades K-6 

2. Project Conquest . East St. Louis, Illinois. 

Reading, grades 1 (repeaters) -6 

3. Programed Tutorial Reading , Davis County, Utah. 

Reading, grade 1 

^' Intensive R,eading Instructional Teana , Hartford, Con- 
necticut. Reading, grade 3 

5» High Intensity Tutorlni^ . Highland Vaik, Michigan. 
Reading and math, grades 6 and 7 

6' Project R-3 . San Jose, California 
Reading and math, grades 7-9 

Some 2,000 projects had been considered at least superficially, 
136 appeared to be possible candidates at some point during the screening 
process, and 103 were reviewed in depth. The extent of the searching and 
selection task far exceeded what had been planned. Additional details are 
provided elsewhere in this report and in Foat (1974). 



Some preliminary planning regarding packaging concepts was also in- 
itiated very early in the study^ This effort began in earnest^ however, 
as soon as the first exemplary project was identified. Considering the 
specific information potential adopters would need to have in order to 
replicate key project features proved to be a particularly fruitful way 
to proceed* 

The development of packaging concepts and the first Project Infor- 
mation Package proceeded simultaneously » imdergoing many iterations as 
new ideas emerged or old ones proved unsatisfactory* Approximately four 
and one-half ninths after work on the contract began, an External Advisory 
Panel was convened to review and evaluate packaging concepts and draft 
materials. Representatives from several groups in O.E. and the National 
Advisory Council on the Education of Disadvantaged Children were also in 
attendance* 

The two-day meeting produced many useful ideas and led finally to 
the specification of the components and subcomponents which the Project 
Information Packages (PIPs) would contain as well as the manner in which 
they would be mediated. These specifications were called the Model Design* 
Ideas were also obtained for an ^Xnalysis and Selection Kit which would 
enable potential replicators to match the needs of their schools with 
the characteristics of the packaged projects. 

The actual packaging of the PIPs was done by Learning Achievement 
Corporation of San Jose, California under subcontract. Because of this 
arrangement, it was necessary to prepare detailed specifications for each 
of the projects selected for packaging. These specifications, called 
Project Models, were organised in accordance with the Model Design referred 
to above but included detailed information about the content, emphasis, 
and method of presentation to be reflected in the final PIP con^jonents 
and subconq>onents* While the first two PIPs were developed without formal 
Project Models, such models were prepared and used for the four remaining 
projects. Further information about the Model Design and Project Models 
is given later in this report and in Piestrup (1974) . 

The PIPs themselves are comprised of nine components, four for 
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planning and five for implementing each project* The components include: 

1. Starter Set; Planning . Provides orientation to project, public 

relations material , and introduction to the package for 
the project director. Also provides information for school 
boards, principals, regular school staff, potential school 
staff, and parents, in order to inform and elicit support. 
It includes a filmstrip with cassette tape, handout brochure, 
project director's orientation booklet (with cassette tape for 
some projects), and viewfoils. 

2. Project Management Directory . Provides detailed guidelines and 

support materials needed to plan and implement the project for 
project management personnel. It includes a Project Management 
Calendar and supplementary sections on the major management tasks. 

3. Project I'lanagement Displays . Provides time schedule overview, 

summarises component vse, and displays component use and time 
schedule to visitors. It includes a Major Managemient Tasks Chart, 
and Project Information Package Use displays. 

4. Staff (Salifications and Preparation Set . Provides personnel sel- 

ection guidelines for director and self -evaluation and self-train- 
ing materials for the staff. It includes a Staff Qualifications 
Checklist and In-service Training Topics. 

5. Starter Set t Implementat ion . Helps classroom personnel in starting 

each type of activity (testing, teaching, other) including setting 
up environment for the first time. It includes Implementation 
Starter Booklet and Original Art File. 

6. Classroom Management Directory . Provides detailed guidelines for 

all classroom procedures and samples of materials needed for 
administration of the project such as forms, notices, and letters 
for project classroom personnel. It includes Teaching Staff 
Guide (e.g., a calendar and support materials). 

?• Student Relationships Album . Tor the project staff interacting 
with children, this album conveys the project environment, from 
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* the child *s viewpoint » which staff is expected to create, e.g^t 
how he should perceive staff » what learning clitoate he should 
experience (high pressure » self-directed, etc.). It also distin- 
guishes roles of different staff ©embers in creating the environ- 
ment and describes desired student responses (e.g., confident, 
competentt happy* eager) and gives specific instances. 

8* Professional Relationships Guide . Defines for all project staff 
(plus the school principal) their roles in relation to all school 
staff (project and non^-project) with whom each project staff mem- 
ber interacts. It also attempts to anticipate and reduce staff 
conflicts. 

9. Hardware/Software Packet . Aids project director and teaching 

staff in selection and ordering of commercial hardware/software 
and describes (including sources and features) core and supple- 
mentary items* Conveys to the teaching staff the experience of 
original project staff plus modifications made (if any). It 
includes Factsheets and manufacturers^ brochures for core items 
selected by project staff and a supplemertary materials list with 
publishers' addresses* available Factsheets* and brochures. 

Each PIP is undergoing extensive demonstration and testing at from 
two to five sites during the 1974-75 and 1975-76 school years. Results of 
the deuKJnstration* based on intensive monitoring and evaluation, will be 
used to revise the PIPs. 
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II. ESTABLISHMENT OF PROJECT SELECTION CRITERIA 
AND VALIDATION PROCEDURES 



The following pages deal in some depth with the criterion areas of 
availability, cost, replicability, and effectiveness (statistical sig- 
nificance and educational significance). Other equally relevant criteria 
were enumerated on pages two and three of the preceding chapter. Since 
these criteria are self-explanatory, however, they are not discussed 
further here. 

All of the project selection criteria were directly related to the 
issue of whether projects, when replicated at new sites, were likely 
to be cost-effective means of helping children learn basic skills. Pro- 
cess criteria other than those required by D.H.E.W. regulations and policy 
were specifically excluded from consideration. In particular, a project 
was given no preference because of good evaluation design or systematic 
development procedures, although these factors substantially facilitate 
project validation and might contribute to Project Information Package 
development. Neither did a project receive preference for meeting its 
own stated objectives unless, of course, these objectives included achieve- 
ment benefits in reading or math. In short, projects were judged on 
results, not on how closely they followed currently endorsed development 
procedures. 

The process of selecting and validating projects was iterative 
in nature, with each criterion area examined at several preliminary levels 
before analysis was undertaken to the depth which was ultimately required. 
As soon as it became clear that a project failed to meet one of the cri- 
teria, it was excluded from further consideration. While this approach 
precluded a detailed analysis of deficiencies (since multiple deficiencies 
would not then normally be detected), the economies it afforded were 
absolutely essential to completing the task in a reasonable amount of time. 

The most technically difficult problems to be dealt with arose in 



tha effectiveness areat for which a 23-step ^'decision tree^* was developed 
to structure the validation process* The decision tree was designed to 
reflect the '^factors Jeopardising the internal and external validity of 
eKperiments*' enumerated by Campbell and Stanley (1963) as they related 
to the types of evaluation studies encountered. It also reflected other 
important cons iderat ions , such as the type of scores on which statistical 
operations were performed (raw, standard , scale , percentile , grade*^quiv- 
alent)^ whether comparisons were made against control groups or were 
norm-referenced > and the basis on which experiJsental-^control (or norm 
group) comparisons were made (post test scores » gain scores , covariance 
analysis, etc»)t 

The process of validating a project typically began with an Incom- 
plete collection of reports » da(;at and promotional literature received 
in response to a written or telephoned request » Wlimowlng nhis infor- 
mation » identifying and obtaining needed supplementary data, and weigh- 
ing the resulting evidence was a coisqplex task« It required a substantial 
Investment of effort, including mail and telephone communication with 
project personnel followed by on-site visits. Because this process was 
not feasible for all of the projects which were nominated as candidates 
for packaging, some preliminary screening was required. Projects which 
passed the preliminary screening criteria under each heading were con- 
sidered "possible** candidates for validation and all criterion areas 
were then systematically Investigated in greater depths Projects which 
failed to meet any of the preliminary criteria were not rejected immed- 
iately, but attention was focused on the specific criterion in question 
so that definitely unsuitable projects could be identified and rejected 
with a minimum of superfluous effort. Brief discussions of each of the 
criterion areas follows. 

Availability 

Availability, in the broad sense, may be construed to encompass all 
factors relating to the ability of the adopting school to obtain all 
components required to replicate an effective project* For the purpose of 
identifying exemplary projects, however, a separate criterion area 
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labeled "Replicabillty" was also established. Thus the connotation of avail- 
ability, as used here, is limited to the two specific categories of acces- 
sibility and acceptability. 

Accessibility was construed to reflect considerations of whether a 
project existed in a form that could be visited for the purpose of val- 
idation, whether the responsible program personnel were willing to coop- 
erate to the extent necessary for the development of a replication pack- 
age, and whether documentation of procedures, results, and costs were 
available. All projects were considered to pass initial screening on 
accessibility unless there was clear evidence that one or more of these 
considerations was not met. 

Acceptability was taken to n^an the conformity of the project to 
Office of Education policy on dissemination. It was assumed for purposes 
of initial screening that projects which were operational in public school 
districts met Office of Education dissemination criteria unless they con- 
sisted primarily of a single commercial product. Such single commercial 
products were not considered for validation. 

All of the above criteria were considered iteratively as the valida- 
tion process progressed. Obtaining access to project sites and project 
information was an integral part of the validation process. Promising 
projects were not eliminated on the basis of accessibility, however, 
until all reasonable approaches to pursuing validation attempts had been 
made. 

The validation of project acceptability consisted of identifying 
and documenting project content and components, and submitting these 
descriptions to the Office of Education for approval by the Dissemination 
Review Panel. 

Cost 

Educational cost accounting is a complex process, and it is frequently 
difficult to isolate the costs of an experimental project. It is even 
more difficult to summarize cost figures in a way that will be meaningful 
to schools in other regions, because (a) schools start with different 
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existing resources » so that a given component of a project may involve 
special expenditures in one district while the same component may be 
available at no cost in a second district » (b) the methods and costs of 
providing prerequisites for success (e.g.» adequate nutrition, discipline) 
depend heavily on pre-existing local conditions » (c) the costs of specific 
items including personnel^ materials^ equipment^ facilities^ etc., vary 
from region to region, (d) both total and per^^pupil costs may vary widely 
as a function of the niunber of pupils served and the efficiency of the 
particular project configuration in a particular setting, and (e) account- 
ing procedures and systems differ widely. 

In evaluating potentially useful projects, the major cost consider- 
ation for most school districts is the additional funding that will be 
required in order to install and operate a program* For the reasons cited 
above, certain assumptions had to be made before any reasonable estimate 
of these costs could be obtained* The most significant of these assump-* 
tions was that adequate facilities to house the project could be made 
available at no cost to the project. If, for ex^itple, it became necessary 
to purchase a portable classroom, such costs would almost certainly exceed 
any reasonable selection criterion which could be established* Another 
significant decision was that projects would be ''costed^' in an efficient 
configuration regardless of the manner in which they were being implemented 
at the originating site. Finally, since whatever cost criterion would be 
agreed upon would be highly arbitrary, a decision was made to use local 
cost data without attenq>ting to adjust for regional differences in salary 
or other costs* 

Based on these assumptions, projects which required yearly expendi- 
tures of over $400 per pupil or an initial investment of more than $1,000 
per participating pupil would be rejected. The former figure was subse- 
quently modified upward to $475 per pupil. 

Judging from cost figures reported by a wide range of Title I and 
Title III projects, these costs would not be prohibitive in school 
districts receiving federal funds for compensatory education. It would 
remain, of course, for each adopting school independently to evaluate the 



feaslbilicy of a given project and to determine a realistic cost estimate 
with the help of a needs and resources assessment. 

Replicability 

^fany of the obstacles to replicating a project were considered under 
the headings of Cost and Availability above. This section deals with 
two remaining categories of replicability considerations* One is the 
extent to which the basic model for a project lends itself to packaging 
or reproducing the individual components of the fl»>del for a reasonably 
wide range of potential project users. 

Ease of packaging is primarily a function of the degree of structure 
in a project and the comprehensiveness of its documentation. Highly 
structured projects in which the learning process is controlled largely by 
printed materials or mechanical devices are relatively easy to model, 
whereas t in projects which give teachers a major role in directing learning, 
it may be difficult to isolate the essential components. In either case, 
projects which have concerned themselves with dissemination may have 
already made substantial progress toward developing Project Information 
Package content. 

Ease of implen»3ntdtion is affected by both the unfamilidrity and Che 
complexity of teaching methodology, management, scheduling, and data 
processing components of the system. Projects which require school 
personnel to understand and apply new principles ii!qK>se special selection 
and/or training requirements. Additional problems are inherent in pro- 
jects which involve the family, the community, or other groups outside 
the direct control of the school. 

The second category of replicability considerations is comprised of 
problems in duplicating essential components of the educational project. 
All components including personnel, materials, machinery, and the gen- 
eral environ<T!ent must be referenced. The only issue considered here was 
whether each component would be available to a prospective user, pre- 
simably without the development of major manufacturing or training institu-^ 
tions. 
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For purposes of initial screening, all operating projects were 
considered repllcable unless a major component could not be readily dup- 
licated. While specialized materials or facilities and expensive or out- 
of-productio(i hardware were obvious problen^* special attention had also 
to be paid to unusual qualities (e.g., charismatic leadership) or quallfi** 
cations required of personnel, and to unique environiaents (e.g., univer- 
sity settings) that could not be obtained or constructed by a school dis- 
trict. 

In the final analysis, decisions as to whether or not a project was 
adeqitately repllcable might have rested on highly subjective judgments 
Only a few projects were rejected on the basis of replicabllity consid- 
erations, however, and these few cases seen^d clear cut. 

Effectiveneas 

Assessing the effectiveness of an educational project presents an 
intrinsically difficult problem. The evaluator faces many pitfalls 
which may be broadly grouped into the three categories of measurement, 
experimental design, and statistics. Hazards exist in each of these 
areas which s^y completely invalidate any inferences he might draw 
about project intact. 

Conventions for experis^ntal design and associated statistics have 
been developed to deal effectively with evaluation problems in controlled 
experimental settings. Standard reference books describing these con- 
ventions are widely available (e.g., Winer, 1971) and are well known 
to most evaluation specialists. Unfortunately, in the real world of 
education it is often io^osslble to employ rigorous techniques, and It 
is extremely rare to find a compensatory education project which satid«- 
fies all, or even most of the fundamental principles of good research 
design. The problem is so widespread, in fact, that if one were to 
reject all projects with iess-than-ideal evaluations, the possibility 
of finding even a few exemplary projects would be extremely remote. 
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Many of the weaker designs have been discussed at length by 
Canpbell and Stanley (1963) along with the "threats to Internal and 
external validity" associated with each. These authors, however, have 
hardly touched upon the related problems of educational measurement* 
Scoring, scaling, and normlng considerations become particularly Im- 
portant in those designs which employ non-comparable comparison groups 
or no comparison group at all. 

The extent and complexity of the experimental and measurement 
problems made it clear that a systematic procedure was sorely needed 
for reviewing project evaluations, for identifying and assessing the 
Impact of their shortcomings, and for making reasonable jud^nts 
regarding project effectiveness while carefully weighing all relevant 
factors. To meet this need, a 23-8tep decision tree was developed* 
The decision tree was designed to insure examination of each of the 
12 threats to valid inference discussed by Campbell and Stanley (1963) 
as they relate to specific evaluation designs. It also encot^asses 
other Important considerations such as the type of sipnea on which 
statistical operations are performed (raw, standard, scale, percentile, 
grade'-equivalent), whether comparisons are made against "c»>ntrol groups 
or are norm-referenced, and the bases on which experimental~control 
(or norm group) comparisons are made (posttest scores, gain scores, 
covarlance analysis, etc.). 

A procedure of this type cannot, of course, be applied In a vacuum. 
It must be tied to pre-escabllshed criteria to which each judgment can 
be related. These criteria Include (a) the mlnltcum Increment of cog- 
nitive benefit which will be considered educationally significant and 
(b) the minimum non-chance probability level which will be accepted as 
statistically significant. 

It should be pointed out that the establishment of criteria for 
educational and even statistical significance Is a matter of policy 
decision-making and has only tenuous ties to "science." There are 
associated measurement problems, however, which represent scientific 
challenges of a non-trivial nature. Most educators, for example, will 
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agree that the goal of compensatory education is to raise the achievement 
levels of disadvantaged children from some starting point to an end point 
which is closer to the national norm. The quest ions » **How much closer?*' 
must be answered by the policy makers. Once this criterion has been 
agreed upon, however » the problem of how to n^asnre the improvement must 
be re solved « 

Clearly » the only fair measure of an "educationally significant 
gain*' would be one which was independent of the pre-treatmei?t achievement 
atatus of the project participants. In other words » a project serving 
severely disadvantaged children should have just as good a rhance of 
being judged successful as one serving children closer to the national 
norm. To achieve this goal, the criterion of success must be defined 
in terms of an equal-interval scale with some sort of michor point. 
Normalised standard scores referenced to a national average appeared , 
to offer the most appropriate medium in which such a criterion can be 
cast since using unstandardized and/ox criterion-referenced tests 
would require that success be defined in some other manner, and there 
could then be no assurance of equitability over the entire range of 
initial disadvantagement . 

These conditions led the research team to advocate a definition 
of educational significance which was expressed in terms of standard score 
gains referenced to the national norm. A gain of one-third standard 
deviation was subsequently agreed upon with O.P.B.E. as the criterion 
to be used for determining exemplary status. Under these conditions, tor 
a project to be considered for packaging, the mean posttest standard score 
of project participants had to be one-third standard deviation higher with 
respect to the national norm than the mean pretest score of the same 
children. The criterion adopted for statistical significance was the 
five percent confidence level. 

As mentioned earlier, a 23-step decision tree was developed to 
provide a systematic validation procedure for determining whether pro- 
jects met the established effectiveness criteria. The tree itself is 
shown in Figure 1. The report in which it was originally prefiented 
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Fig. 1. Decision tree for validating statistical significance. 
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(Tallmadge and Horst, 1974) also contains in-depth discussions of each 
step as well as technical appendices covering additional relevant details. 
Many, if not most of the steps in the decision tree explicity call for 
judgments from the evaluator. At each step it is assumed that the eval- 
uator is thoroughly familiar with the issues involved and is qualified 
to make a judgn^nt based on complex technical considerations. 

The particular path to be followed through the decision tree depends, 
of course, on the specific design employed in the evaluation study under 
consideration, but each path is structured so as to focus attention on 
the design, analysis, and interpretation pitfalls likely to be encountered 
using that model. Unless a project has been evaluated in several dif- 
ferent ways, substantially fewer steps will be required than the 23 which 
comprise the entire decision tree. 
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III. IDENTIFICATION OF EXEMPLARY PROJECTS 



Originally 9 It was not the intent of this study to undertake a 
wide-scale search for successful projects similar to that which charac- 
terlEed several recent research efforts sponsored by U.S.O.E. (e.g., 
WargOt CaiqpeaUt Tallmadge, 1971). Candidate approaches were to be selec- 
ted from aixu>ng those identified as exemplary by other investigators. An 
initial list of twenty-three promising approaches was provided by the U. S. 
Office of Education at the onset of the project. It was assumed then that 
moslt if not all of the projects to be packaged would come from that list. 
Unfortunately 9 only three met the established criteria , and the search had 
to be extended to include other lists of exemplary project nominees from 
state- and federal-^level a^ncies« projects included in the 1971 and 1972 
Ed Fairs t and projects identified through personal professional contacts. 
It is estimated that, prior to the selection of the final six projects , 
the original list of twenty-three grew to well over 2,000 projects. 

The initial screening process began with a weighing of each cand*^ 
idate project against the prerequisite criteria of grade level, content 
area, target population, and number of evaluation "Instances.*^ Briefly, 
a project was considered a "possible*^ candidate for packaging if (a) 
its approach was aimed at producing reading or math benefits, (b) if its 
approach was used with "target" children (as defined within the guide- 
lines of Title I) In grades K through 12, and (c) if the approach was 
evaluated more than once. In most cases, information from the original 
nomination source centered around content area and grade-level informa- 
tion, with brief projoct descriptions and one-* or two-line statements 
of the evidence of success and availability. Projects which clearly 
did not appear to meet the prerequisite criteria were eliminated from 
further consideration. Occasionally, enough information was available 
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at this juncture to eliminate projects for other reasons as well. Some, 
for example, were easily recognizable as single commercial products and 
others were clearly not U.S.O.E. --funded. Of the approximately 2,000 
approaches reviewed, 136 remained in the study after this initial 
screening and were considered viable candidates for packaging* 

Contacts were initiated with all of the 136 candidate projects 
either by mall or telephone in order to request additional Information. 
Follow-up contacts were made to those projects which failed to respond 
inlLially. Useful information was received from 103 of the projects 
contacted. Four other projects also responded but only to indicate that 
they had been, or were about to be terminated or that no project infor- 
mation was available. No responses were obtained from the remaining 
twenty-nine projects^ although four of these had Indicated that infor- 
mation would be forwarded. 

As promotional literature, descriptive infonnationt and/or evaluation 
reports were collected, each of the remaining 103 projects was reviewed 
by one or more members of the research staff. The initial review en- 
tailed reading the evaluative and descriptive literature and noting 
which of the established selection criteria appeared to have been met 
and which were not met by the project. Often it became clear at this 
point that projects were not suitable for packaging. 

Projects which survived the initial screening were subjected to a 
further in-depth review and analysis. Unfortunately, the Information 
available on projects was typically Inadequate and inconclusive. It 
was often possible to reject projects with a high degree of confidence, 
but those not rejected could only be categorized as "likely*' or "low 
priority" (unlikely) . 

Projects which were considered "low priority" were those which 
appeared to meet the preliminary screening criteria but which presented 
one or more evaluation or packaging problems. In many instances, data 
were Inconclusive, not particularly promising , and would have required 
massive reworking before valid inferences could be drawn. Other projects 
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were placed in the low priority category because charismatic leadership 
appeared to be essential to their success, because the economic interests 
of project developers were threatened by the type of packaging to be 
undertaken > or because the nature of the project suggested the need for 
excessively expensive packaging techniques or media* Each low priority 
project was reviewed in considerable depth by the research staff , and 
every effort was made to retain as many projects as possible* 

Efforts to obtain additional information were initially focused on 
the apparently most promising of the projects ♦ and new inf ormat ion was 
reviewed as quickly as it was received. Unfortunately, it was never pos<" 
sible to accept the evaluation results for any project at face value. The 
main source of difficulty was the almost universal use of grade-equivalent 
scores and grade-equivalent gains, but other problems such as scoring 
irregularities were also encountered. Before a ny project could be accepted 
as successful it was necessary to obtain the raw test scores and to con- 
duct appropriate analyses on them. The amount of reanalysis required was 
still another factor which original planning for this study had grossly 
underestimated. 

Eventually, all projects in the "likely** a n d **low priority" categories 
were examined in depth. Surprisingly, only six projects were found which 
met the established criteria. Original plans called for the packaging 
of eight exemplary projects. There was thus a significant reduction in 
the scope of the packaging effort itself which served to offset the gref^tly 
expanded effort that had been required by earlier contract tasks. 

In all, project selection was a continuous activity throughout the 
first eight 0K>nth8 of the contract period. The initial selection, Project 
Catch-^Up , was chosen during the third contract month. Hig h Intensity 
Tutoring , Project Conquest » and Pr p gram e d Tutoria l Reading were selected 
during the fifth contract month. Project R-3 and Intensive Reading 
Ins t rue t ional Teams were selected during the seventh and eight contract 
months, respectively. Each of these projects was written up for, and 
approved by* the U.S.O.E. Dissemination Review Panel before the end of the 
ninth contract month. 

Information about the 103 projects which were seriously considered 



as candidates for packaging^ and the reasons why they were rejected, has 
been provided by Foat (1974) • Table I summarizes the frequency of re- 
jections for each of the established criteria. In examining Table I» 
it should be remembered that rejected projects were examined only until 
a reason was found for rejecting them. Individual projects might have 
failed to meet several criteria » but no attempt was made to look for 
multiple deficiencies. The summary data, for this reason, do not nec- 
essarily reflect the frequency with which specific deficiencies exist. 
What they reflect is solely the frequency of occurrence of '•first-noticed" 
deficiencies. 

As indicated in Table I, over half of the candidates for this study 
were rejected for failure to m^et the effectiveness criterion* This 
reason for rejection must not be taken as an indication that the projects 
were unsuccessful — or even that they failed to produce cognitive achieve- 
ment benefits. As is stated in the Tallmadge and Horst (1974) report, 
"What is rejected is not the project but the evaluation data which, if 
the decision-tree process has been carefully followed, have been shown 
to be inadequate as a basis for reaching any conclusion regarding project 
effectiveness [p. 11}." 



22 



TABLE 1 



Summary of Reasons Projects Were Rejected 



Reason for 
Rejection 


Frequency 


Percentajge 


Effectiveness 


52 


S4I 


Relevance 


17 


18% 


Availability 




11% 


Bilingual 


8 


8% 


Replicability 


2 


2% 


Conforaance to 
Guidelines 


2 


2% 


Cost 


2 


2% 


U.S.O.E. Support 


2 


2% 


Follow Through 


1 


1% 


Total 


97 


100% 
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IV« DESIGN OF PROJECT INFORMATION PACKAGES 



Neither the replication of exeioplary compensatory education projects 
nor the packaging of descriptive infonoation about such projects was a 
new idea at the time the irork described herein was begun. What was new 
was the concept of designing a package so that it would itself constitute 
a viable and at least nearly sufficient mechanism for project replication. 
In essence, what was desired was a comprehensive ''blueprint" which would 
enable an adopting site to replicate at least the essential features of 
a defiK>nstrably successful project with sufficient fidelity to ensure a 
high probability of success in the new setting — and with a minimum of tech- 
nical assistance. 

The developers were aware that earlier attempts to bring about the 
replication of effective educational practices had been only marginally 
successful. Asu;>ng the reasons for this disappointing state of affairs 
are the following: 

1. Descriptive data of the type usually generated for successful 
or promising projects are not pad ^ged effectively; most fre- 
quently there is a demand for a good deal of dedicated reading 
on the part of the potential program adopter. FurtheniK>re » 
the material does not guide the adopter through the day-by-day 
problems and objectives of £he successful project. 

2. Wnen the printed materials are edited dovm, and even when 
they are cosdl>ined with visual aids, they do not communicate 
the real impact and the presence of the successful or model 
projects. 

3. Prospective project adopters are basically n^st interested in 
procuring for their schools the critical two or three elements 
of a model project that account for most of the model^s success. 
But they are most often presented with a shopping list that 
suggests neither priority nor ranking for the listed items. 

if 
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4. Adoption techniques are not usually identified* If a teacher 
or administrator shows interest in replicating a program, he 
is given elements of the project without instructions for re*- 
applying those elements to a new setting. 

The avoidance of these pitfalls becan^ one of the major considerations 
in conceptualizing Project Information Packages* 

It was felt that a major improvement over previous attempts would 
result from the integration of management functions with the instructional 
paradigms of the projects selected for replication* This approach would 
contrast markedly with the standard approach which simply described an 
operating program without consideration of what must be done and by 
whom in order to install and maintain the project. 

Consistent with this "how-to-^o-it** orientation, it seemed appro- 
priate to try to arrange the n^terials in son^ sort of ti^-based se* 
quence. VRiile the coiq>leKity of the projects, and the ntunber of dif* 
ferent persons involved in their installation and operation precluded 
organisation based entirely on time phasing, it was at least possible 
to group the components into planning and implementation groups. As 
far as possible, components within groups and materials within components 
were also arranged according to the sequence in which they would be used 
in the replication process. 

These derisions were arrived at early in the contract period. It 
was not long thereafter when the general configuration of PIP components 
began to take shape. The niuaber, nature, and content of the components 
changed more or less continuously, however, as development of the first 
PIP progressed. Concepts which sounded good in theory simply could not 
be Implemented satisfactorily, and new ideas emerged to replace them. 
What is described below is the end result of what was really an empirical 
tryout and revision process. 

Planning Cogg>onents 

The projects selected for packaging had typically operated for 
several years and had undergone a variety of evolutionary changes. In 

25 



all cases, It was clear that the original start-up procedures would be 
less than optimum replication strategies"***in some instances ^ dramatically 
so. But devising strategies for installing full-^blown projects and 
getting them to operate smoothly the first year posed severe problems 
as well* It was necessary to invent a planning phase, scheduling events 
by means of logical inference from known end points* For example, to 
have personnel, instructional materials, and space available for a 
laboratory in September, it could be inferred that the activities of 
hiring teachers and aides, ordering equipti»3nt and supplies, and arranging 
^for space with a principal must begin some months in advance of September. 
These events had further implications* If equipment is ordered, there 
Qmst be a place to have it delivered; if space is needed in a school, the 
principal must determine how and where to provide it; if the regular 
school staff is to be involved, their concurrence and that of the prin** 
clpal must be obtained; and so on* Thus, events bad to be identified 
and a schedule worked out which \^uld be consistent with task interde- 
pendencies, lead-time requirements, and desired (or required) conq;>letion 
times* 

One of the first decisions in developing the PIP design was that 
key events, particularly for the planning phase of the project, should be 
scheduled in a calendar format* The concept was to list tasks in the 
calendar which were to be accomplished as close to when they were scheduled 
a» possible* How they were done was to be left variable* Support docu<* 
ments with descriptions of how the original project accomplished the task, 
Ineernal memos that showed the style of management, alternatives for 
dealing with difficulties, and other practical suggestions were to be 
referenced in the calendar* Events were to be blocked by weeks so the 
director could fill in the actual event, such as a meeting with his or 
her principal, on the day it was to occur* The directory was also to 
summarise the tasks for each month in a checklist format to emphasize 
the importance of accomplishing them as nearly as possible to the scheduled 
time* The directory was also to provide budget updating summaries for the 
director to fill in each month* This key component was named the Project 
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Manage ffBnt Dire ctory (PIP Component ff2). 

The first f^w weeks of the replication process pose special problems 
in the area of orientation and public relations. A Starter Set ; Planning 
(PIP Component #1) was therefore called for to give the new project director 
fmilti*^dia materials to use in presenting the project and in gaining the 
support of parents, teachers, principals, and school boards. The purpose 
of the Starter Set is to explain the key features of the project briefly 
for a general audience, and in some depth for the project director so that 
he or she w^xxld be able to conduct subsequent briefings and answer ques- 
tions on the project. 

Brief one-page handouts and illustrated brochures were also specified 
for the project director to use in public relations as part of the Starter 
Set : Planning . In the course of the project, the director and staff would 
probably develop their own brochures and notices to send home but, again, 
the PIP model was designed to accelerate the start-up process by calling 
for materials the new staff would need in a format designed for immediate 
use. 

Pro;!ect Management Displays (PIP Component 03) were also called for 
in the model design to serve a public relations and infommtion function. 
They were to be designed to attract attention to the existence of the pro^ 
ject and to elicit questions or interest concerning it. They also were 
to summarize the key tasks to be performed and the components to he U2»ed 
in performing them. 

Specifications for a final planning component, the Staf f Qualifica- 
tions and Preparation Set (PIP Component H) were drawn up to assist pro- 
ject directors in hiring the kinds of personnel needed to implement speci- 
fic projects and in providing training and training mechanisms for project 
teachers and aides. Because major differences were found between projects 
with respect to the relative emphasis placed on hiring as opposed to train- 
ing (the more highly structured projects tended to rely on the hiring of 
existing expertise), two separate sub-con^onents were called for— one re- 
levant to selection and one to training. The PIP model design specifica- 
tion which was eventually prepared suggested that the two sub-components 
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should be equally detailed. 

In the training area» the m^del design did not call for the devel^ 
opment and/or nMsdlation of actual Instructional materials » Rather » it 
specified that training programs be defined. The assumption was that 
project directors would possess* could acquire* or could hire outside 
experts to provide the required subjects-matter expertise. 

These four coc^nents were designed to provide all that would be 
needed to plan & project from the time a district decided to adopt it 
until it began to operate in the schools. Most of the operational direc- 
tions were to be contained in the Project >^na8ement Directory , but other 
components were to provide further information and mediation of the pro- 
ject designed to facilitate smooth and efficient Introduction of the 
project into replicating districts. 

Ifltpleinentation Cot^on e n ts 

Project implementation components are those related to instruction* 
including teacher orientation* classroom management, working with pro- 
fessional associates* and the purchasing of materials for instruction* 
First* it was decided that the teachers and aides should be provided 
with an introductory component analogous to th€ Star ter Set for the 
project director. The design assumed* however* that teachers would 
have been briefed on the project at the time they were hired and would 
have gone through in'-service training before using the Starter Set ; 
Iflg>lementation (PIP Component #5). This con^onent was created to pro- 
vide extra help in the early weeks of instruction in the new project. 
The design called for materials for decorating the classroom* a detailed 
calendar for the first two weeks of school* and descriptions of how to 
begin each new activity* whether it was testing or using &i>ecial equipment. 

Specifications for this comporent changed considerably during the 
development process. At first* the intent was to provide a detailed 
lesson plan for the first few hours or days of instruction. It was 
later learned, however* that most projects selected for packaging were 
unlike traditional classrooms where teachers would face thirty new faces 
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the day after Labor Day. Working groups were typically sioall, the first 
few weeks were generally devoted to testing, and neither teachers nor 
children were faced with an abrupt beginning or all-day large-group inter- 
action. 

Despite these mitigating factors, it did seem in^ortant to provide 
infonsation and some materials on duplicating whatever environnwantal 
conditions the project director considered essential for motivating 
children from the start. Again, the essential idea was to enable new 
staff to create an attractive environment appropriate for the region 
where the project was to be replicated, not to package the particular 
bulletin boards or decorations used at the originating site. This 
component was intended to provide a place for including some actual 
materials for teachers to use, but these materials were to be modified 
so as to be useful in a variety of contexts other than the one for which 
they were originally designed. It was also to draw from the ideas of 
originating teachers on how to make devices useful in the laboratory 
setting e.g., how to make study carrels, how to nake bright, round 
fables out of old desks, and the like. The idea was not only to share 
these ideas but also to encourage new teachers to approach the project 
with inventiveness and commitment similar to that of the originating 
teachers. 

The Clas pr opm Management Directory (PIP Component ^6) was designed 
to correspona to the director's calendar in format *ad purpose, but with 
a day-by-day emphasis on guidance and format. Calf jar entries were to 
indica.ce the sequence of events and to remind teach, rs to perform key 
t8sk6 throughout the year. Some tasks, such as scheduling time for 
children to be released by regular teachers, were continuous over several 
days or weeks. Others, such as pretesting, needed to be performed on a 
certain day. The Classroom Management Directory was also supposed to in- 
clude monthly task summaries in a checklist format, budget records, and 
supplementary sections explaining alternative strategies for accomplish- 
ing tasks and anticipating problems. Thn i>rsctical details of operating 
an instructional system were to be described in this component, with 



29 



information drawn from the people who successfully taught in the orig- 
inal project. 

The components described thus far have dealt with how the program 
should be set up and operated. It was also considered extremely im- 
portant to describe the project from the child* s viewpoint, and the 
Student Relationships Album (PIP Component //7) was designed to fill this 
need. The roles of each staff member in relation to the child were to 
be included on information sheets. In addition, space was to be provided 
for recording information on new products along with a list of publishers 
of educational material 

Project Models 

Once the overall configuration for the PIPs had been finalized and 
the characteristics of each component defined, the next step in the devel- 
opment process was the preparation of detailed models for each of the pro- 
jects to be packaged. These n»dels were carefully detailed documents 
which described the exact content, the method of presentation, and the 
form of mediation for each component and subcomponent of each PIP. All 
models were developed in accordance with the basic PIP model design and 
thus bore at least superficial resemblance to one another. The similar- 
ity, however, did not extend beyond organization and format. The content 
of the models differed greatly and reflected, of course, the differences 
which existed among the projects selected for packaging. 

It was even necessary, in some instances, to deviate somewhat from 
the model design. In one model, for example, the role of the school 
principal was critical while the roles of the actual instructional person 
nel were entirely "programed". There was no need for a Classroom Manage 
tnent Directory (Component ilb) whatsoever, and it was replaced with a Pritv 
cipal's Management Directory which was quite different in content and 
format from the specifications of the model design. This same model also 
called for the inclusion of some self-instructional training materials in 
Student Relationships Alburn^ — another departure from the model design. 

The development of models for specific projects involved a great 
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deal of Information gathering* All available doc mentation was reviewed 
and, once projects were finally selected, site \ • Its were conducted. 
Four to six man*-days were typically devoted to sitt visits although two 
projects involved substantially more time. Follow-up telephone contact 
was often required to clarify specific points as the tiu^dels were devel- 
oped. 

The first (or prototype) PIP was based on Project Catch-Up and was 
developed without a model through an iterative process involving close 
coordination between RMC Research Corporation and Learning Achievement 
Corporation. An External Review Panel comprised of two teachers * two 
school principals, and two ^'experts in compensatory education" also con- 
tributed significantly to the design process by reviewing and commenting 
on early Ideas and draft materials. A similar role was played by a 
large number of representatives from various offices within the U. S, 
Ot;fice of Education and by members of the National Advisory Council 
on the Education of Disadvantaged Children. 

The first formal review was held in California in mid-November 1973 
and involved all of these groups. In mid-January 1974, a second review 
was held in Washington, D. C, but the External Review Panel did not par- 
ticipate. A third review was held in California in late April 1974, and 
the Exte rnal Review Panel was again actively involved. 

By the time of the second review n^eting, the prototype PIP was 
essentially complete although most of the components were not in their 
final mediated form* Some additional revisions were made based upon 
comments received at the review meeting, but they were relatively minor. 
The third and final review concentrated on draft models and materials for 
the five effective projects still to be packaged • 

Once the prototype PIP was approved, it served as a design guide for 
the subsequent packages • The project models provided detailed specifi- 
cations for content and organization; the prototype PIP filled a similar 
role for details of fonnating and mediation. 

The project models would not have had to be drawn up so formally or 
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in such great detail (a sample model is contained in Piestrup, 1974) had 
a single contractor been responsible for the entire development effort. 
Because Learning Achievement Corporation did the actual packaging, however, 
the project models also served as subcontractor work statements and, to- 
gether with the prototype PIP, provided all the information needed to 
develop the package. 

Scripts and other draft materials prepared by Learning Achievement 
Corporation were revievred for adequacy and accuracy by meinbers of the KMC 
project staff before final taping or typesetting. As far as possible, 
these materials were also reviewed by the project directors at the orig- 
inating sites. 

Two PIPs, in their final form, are pictured in Figures 2 and 3. 
The boxes themselves are approximately 23 1/2 inches long, 14 1/2 inches 
high, and 13 inches deep. They contain ten upright drawers, one for 
each of the first eight PIP components and two for the ninth component, 
the Hardware / So f tware Packet . There is also a large drawer at the top 
of the box for oversiae artwork and materials. The typical PIP contains 
several hundred pages of amply illustrated printed materia? including 
wall-size management displays, one or more filmstr1.ps with accompanying 
audio cassettes, original artwork and/or photographic mater ialj illus- 
trating room arrangements and decorations, and manufacturers* brochures 
and order forms for core hardware and software items. 
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V. INITIATING THE FIEUD TRYOUT 



Quite early in fiscal year 1974, planning was initiated within 
U.S.O.E. for a field tryout of the Project Information Packages. Pro- 
posals from school districts interested in adopting one of the PIP models 
were solicited in spring of 1974 and grants were awarded to 21 sites in 
June. Tw) sites subsequently dropped out. 

The original plan was to test each of the PIPs at three sites, 
but for a variety of reasons— including the two dropouts— the number 
of sites per PIP ultimately ranged from two to five. The following sites 
are currently implementing the PIP projects: 

Project Catch-Up 

Bloomington, Indianna 
Wayne Ci.ty, Illinois 
Providence Forge, Virginia 
Galax, Virginia 
Brookport, Illinois 

Project Conquest 

Benton Harbor, Michigan 
Cleveland, Ohio 
Gloversville, New York 

High Intensity Tutoring 

Lexington, Mississippi 
Olean, New York 

Intensive Reading Instructional Teams 

Oklahoma City, Oklahoma 
Schenectady, New York 
Bloomington, Indianna 
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Programed Tutorial Reading 



Canton, Mississippi 
Dallas* Texas 

Project R-a 

Lake Village, Arkansas 
Lorain » Ohio 

Charlotte, North Carolina 
Schenectady, New York 

The purpose of the field tryout is twofold. The first objective 
is to determine whether, oi to what extent. Project Information Packages 
can serve as a viable means for replicating exemplary educational prac- 
tices. The second objective, assuming that PIPs are viable replication 
mechanisms, is to identify deficiencies in the packages themselves and 
revise them before more general distribution is undertaken. 

To achieve these objectives, U.S.O.E.'s Office of Planning, Budget- 
ing, and Evaluation awarded a contract to the Stanford Research Institute 
of Menlo Park, California to conduct both a process and outcome evaluation 
of the field tryout. RMC Research Corporation will be working with SRI 
over the two-year period of the evaluation study. A report summarizing 
the first year's results is scheduled for publication in the fall of 1975. 

Delivery of the PIPs to the replication sites occurred substantially 
later than would have been optimum. Planning and installation phases 
of the replication process had, consequently, to be substantially com- 
pressed. While this unfortunate situation makes it Impossible to deter- 
mine how well the PIPs would have worked had they been available earlier, 
there is already some evidence that at least some of the PIPs are result- 
ing in successful replication at some of the sites. 
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